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INITIAL STEPS 

Creating a Project Timeline 



Month 1 Month 2 Month 3 Month 4 Month 5+ 



Sample Timeline* 

Launch Project 

Identify district and/or external partner 
working team and develop project plan 

Compile district performance overview 

Conduct Analysis 

Gather necessary data 
Complete analysis 

Review results and iterate as necessary 

Engage Stakeholders 

Communicate purpose of analysis and seek 
support 

Communicate findings and ongoing strategy 

Implement recommendations (detailed in Portfolio 
Development Guide) 






Ongoing implementation 



V 



★ ♦ ♦ ♦ ♦ ♦ ★ 

Kick Off Update 

Meeting Meeting 

The whole process— from analysis to implementation— takes significant time and effort to complete so expectations need to be managed accordingly. 

* Actual time may vary depending on working team composition, personnel dedicated to project, and complexity of analysis and implementation. 
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High School Portfolio Development: 




DEVELOPING BASELINE DISTRICT FACTS: 

Cross-Sectional Analysis 



Key 

Definition 



Crosszsectional analyses use data at a specific point in time or for a 
specific time period (e.g., the high school graduation rate in a given 
year). 



District High School 
Five-Year Dropout Rates, 
2000-06 












Percent Free and 
Reduced Lunch 
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85 % 




Sample District 



Student Ethnicity 




White 

Asian 



Hispanic 



African 

American 



Sample District 



Key Steps 

1 . Create list of metrics that need to be 
agreed upon 

2. Calculate these metrics based on 
the previously defined cohorts 

3. Compare calculated metrics to those 
that already exist for the district 
(e.g., graduation rates) to ensure 
accuracy of methodology and cohort 
definition 



Key Decisions/Questions 

1 . What facts are already known/ 
calculated by the district? 

2. What common beliefs about the 
district need to be validated/refuted? 
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IDENTIFYING EFFECTIVE OPTIONS: 

Geographical Concentrations of Student Need 



Location of Overage Late Entrant ELL Students 
by Home Zip Code, 2005-06 
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Key Steps 

1 . Decide what student need 
indicator is of interest 

2. Use cross-sectional population 
definition similar to that in 
“sizing the off-track population" 

3. Link students to home ZIP codes 

4. Calculate percentage of students 
in total population that display 
indicator of need 

5. Map high schools or programs 
focusing on this population 
against student need to assess 
whether need is being addressed 



Key Decisions/Questions 

1. What indicators of student 
need are of greatest interest? 

2. What schools and programs 
currently exist that serve 
specific populations? 
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IDENTIFYING EFFECTIVE OPTIONS: PREVENTATIVE 

Determining Effect of Structure 
on Specific Students 



Local Diploma 
Honors 



Class of 2007 Four- Year Graduation Rate by 8th Grade ELA Level: 
New Small Schools vs. All Other Schools, General vs. Honors Diplomas 



100 % 




Key Steps 

1 . Use either cross-sectional or 
longitudinal student population, 
depending on questions to be 
answered 

2. Determine what student 
characteristic will be controlled for 
(e.g. p 8th grade test performance) 
and what school models are to be 
compared 

3. Link students to schools based on 
previously established rule for when 
students link to schools 

4. Calculate outcomes for each distinct 
student-school grouping 



Key Decisions/Questions 

1 . Does analyzing school outcomes in 
this way sufficiently account for 
student-level differences? 



Overall Gap 18% 26% 17% 9% N/A 

Honors Gap 4% 7% 6% 3% N/A 



Controlling for specific student characteristics and comparing outcomes can 
provide a more nuanced view of performance without greatly increasing the difficulty 
of the analysis. 



Key 

Learning 
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Graduation Rate 



IDENTIFYING EFFECTIVE OPTIONS: PREVENTATIVE 

Developing Basic Predictive Models 



Likelihood to Graduate for Age-Credit Combinations 




Credits at Beginning of Year 



Key Steps 

1 . Use a longitudinal student 
population representing a single 
cohort of students 

2. For each student, determine the 
number of credits earned each 
year and the enrollment status at 
the end of the year (e.g., still 
enrolled, graduate, drop out, 
transfer) 

3. Calculate the graduation rate for 
each credits-earned group for 
each year in time 



Key Decisions/Questions 

1 . When should total credits earned 
be calculated for a year? Including 
summer school credits? 

2. Which graduation rate should be 
used— 4 year, 5 year, 7 year? 



analyst Guide Supplement H 



High School Portfolio Development: 




Graduation Rate 



IDENTIFYING EFFECTIVE OPTIONS: PREVENTATIVE 

Predicting Graduation Rates 
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Predicted Graduation Rates for Small Schools vs. 
Comprehensive High Schools First-Time Freshmen, 

Class of 2010 



67 % 



62 % 



Small Schools Comprehensive Schools 



Key Steps 

1. Use historical relationship between 
credits earned and graduation rates 
(slide 6) to roughly project 
graduation rates of upcoming 
classes 

2. For each student, calculate the 
number of credits. Based on 
student’s credit accumulation and 
the historical graduation rate of 
students who have earned that 
credit level, assign student a 
graduation rate 

3. Take a weighted average of each 
student to calculate total graduation 
rate 



Key Decisions/Questions 

1 . What variables can be known early 
in a student’s career that increase 
power of simple model? 

2. What factors (i.e., promotion policy 
changes) may compromise the 
predictive power of the model? 
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IDENTIFYING EFFECTIVE OPTIONS: PREVENTATIVE 

Advanced Predictions of Performance: Regression Analysis 



Not all regressions have significant school-level effects, as 
these effects may take several years to emerge. 

Learning Regressions using early outcomes like on-track may have 

negligible school-level factors. 



Student-Level Factors 




School-Level Factors 



• Gender 



• School enrollment 



• Ethnicity (White/Asian vs. African- 
American/Hispanic) 

• Average 8th ELA and math 
proficiency level 



• A series of concentration variables: 

- Percent of students with a Low 
Level 2 or below on either ELA 
or Math 



• Age at entry to high school 

• 8th grade attendance rate 

• ELL status (Y/N) 

• Special education (SPED) status 
(LRE vs. self-contained vs. non- 
SPED) 

• Lack of 8th grade attendance or 8th 
grade test scores 



- Percent of students entering 
9th grade overage 

- Percent of students with 8th 
grade attendance below 90 
percent 

• Tested but not significant: avg. 
teacher tenure, principal tenure, 
general education class size, SPED 
class size, facilities capacity 
utilization 



Key Steps 

1 . Determine outcome variable (e.g . , 
on-track status, graduation) 

2. Decide point in time to associate 
students to schools (suggested: 
freshman year school) 

3. Assess variables that are known 
about students at entny (student- 
level) and ane to be tested 

4. Calculate size and concentration 
variables (school-level) that are to 
be tested 



Key Decisions/Questions 

1 . Does the district have the internal 
expertise to develop a complex 
predictive model? If not, with whom 
can the district partner? 
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Actual Graduation Rate 



IDENTIFYING EFFECTIVE OPTIONS: PREVENTATIVE 

Using Multivariate Regression Analysis to 
Calculate Over- and Underperformance 



Relationship Between Predicted Graduation 
Rate and Actual Graduation Rate, 2007 Cohort 




Key Steps 

1 . Use advanced predictive model 
methodology (slide 8) 

2. Sum the individual student-level 
over/underperformance (actual - 
predicted) to determine school-level 
performance 

3. Identify outcomes associated with 
different school models to determine 
whether school model produces 
systematic performance 

4. Determine level of over/ 
underperformance that warrants 
further investigation 



Key Decisions/Questions 

1. Should over/underperformance be 
based on a single cohort’s outcomes 
orthe combination of multiple? 



Predicted Graduation Rate 
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Average Overperformance 
(Higher Positive is Better) 



IDENTIFYING EFFECTIVE OPTIONS: PREVENTATIVE 

Identifying More Effective School Models 



On-Track Overperformance by School Type, 
2005-06 Freshman Cohort 



0.15 

0.12 

0.1 



Key Steps 

1. Use advanced predictive model 
methodology (slide 8) 

2. Group schools by model type 

3. Average school-level residuals by 
model type to identify models that 
demonstrate consistently strong or 
weak performance 

4. Developing profiles of school models 
may help provide context for the 
outcomes 
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Key Decisions/Questions 

1 . How do different school models 
support target student populations? 
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IDENTIFYING EFFECTIVE OPTIONS: PREVENTATIVE 

Understanding Model Variables under 
District Control 



Variable 


Description 


What change would move the predicted 
graduation rate by 2 percentage points? 


Total HS 
Enrollment 


A high school's official 
grade 9-1 2 enrollment for 
the 2006-07 school year 


A 500-student increase (system avg=- 1 ,500) in the 
total enrollment of a school will cause a decrease 
in graduation rate by 2 percentage points 


Low 8th Grade 
Achievement 


Percentace of students in a 
high school's 2007 cohort 
who scored Level 1 or Low 
Level 2 on at least one 8th 
grade exam (of students 
who took one or both 8th 
grade exams) 


A 5 percentage point increase (system avg=55%) in 
the number of "low 8th grade achievement" 
students in a school will cause a decrease in 
graduation rate by 2 percentage points 


Overage at 
Entry 


Percentage of students in a 
high school's 2007 cohort 
who turn 15 on or before 
12/31/2003 


A 5 percentage point increase (system avg=31 %) in 
the number of students in a school who were 
"overage at entry" will cause a decrease in 
graduation rate by 2 percentage points 


Self-Contained 
(SC) Special 
Education 
(SPED) 
Proportion 


Percentage of students in a 
high school's 2007 cohort 
who were ever SC SPED 
during grades 9-12 


A 6 percentage point increase (system avg=10%) in 
the number of SC SPED students in a school will 
cause a decrease in graduation rate by 2 
percentage points 



Key Steps 

1 . Separate schools from regression 
analysis into overperformers and 
underperformers 

2. Select a performance level change to 
test (e.g. p 2 percentage point change 
in graduation rate) 

3. Identify variables that are 
substantially under district control- 
likely school size and student 
concentrations 

4. Hold all other model variables 
constant while varying (solving for) a 
single variable to understand change 



Key Decisions/Questions 

1 . What variables in the predictive 
model are substantially within 
district control of schools? 

2. How does this analysis relate to 
schools that are “structured to fail”? 
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IDENTIFYING EFFECTIVE OPTIONS: PREVENTATIVE 

Understanding the Effect of Size and 
Concentration on Individual Students 



School 1 

• 400 students 

• Concentrations of 
challenged students 

15 percentage points 
below system 
average 



83% Graduation Rate 

(Effect of 400 Student 
Enrollment Alone 78%) 



% 



The Median Student" 

* Female 

» African-American/Hispanic 

* Avg. 8th Grade Test Scores (2.7) 

» Avg. 8th Grade Attendance (92%) 

* On-Age at Entry to HS (14 yo) 

* Neither ELL nor SPED 



* 



School 2 

• 800 students 

• Concentrations of 
challenged students 

5 percentage points 
below system 
average 



78% Graduation Rate 



School 3 


70% Predicted Graduation Rate 


■ 

School 4 


• 1,500 students 




• 3,000 students 


• Concentrations of 


^1 


• Concentrations of 


challenged students 




challenged students 


5 percentage points 




15 percentage 


above system 




points below system 


average 




average 


70% Graduation Rate 




55% Graduation Rate 



Key Steps 

1 . Select a representative student— in 
the example, the student has the 
median characteristics of all 
students— and determine the 
graduation rate independent of 
school structure 

2. Choose several school structures 
that are representative of the variety 
of schools in the district, with 
different size and concentrations 

3. Adjust model size and concentration 
parameters to calculate student 
graduation rate given different 
school structures 



Key Decisions/Questions 

1 . For representative student, how 
much of a performance gain is 
possible with just structural 
changes? 
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Graduation Rate 



IDENTIFYING EFFECTIVE OPTIONS: RECUPERATIVE 

Determining Differential Effectiveness 



Overage and Under-Credited Seven-Year Graduation Rates 
by Reading Level: Articulated High Schools vs. Transfer Schools 
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Key Steps 

1 . Use either cross-sectional or 
longitudinal student population, 
depending on questions to be 
answered 

2. Determine what student 
characteristic will be controlled for 
(e.g. p 8th grade test performance) 
and what school models are to be 
compared 

3. Link students to schools based on 
previously established rule for when 
students link to schools 

4. Calculate outcomes for each distinct 
student-school grouping 



Key Decisions/Questions 

1. How does student performance 
across key metrics vary by school 
type? 



Improvement vs. 
General HS 



278% 



145% 109% 



111% 137% 
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